Video processing with multiple graphical processing units

ABSTRACT

One embodiment of a video processor includes a first media processing device coupled to a first memory and a second media processing device coupled to a second memory. The second media processing device is coupled to the first media processing device via a scalable bus. A software driver configures the media processing devices to provide video processing functionality. The scalable bus carries video data processed by the second media processing device to the first media processing device where the data is combined with video data processed by the first media processing device to produce a processed video frame. The first media processing device transmits the combined video data to a display device. Each media processing device is configured to process separate portions of the video data, thereby enabling the video processor to process video data more quickly than a single-GPU video processor.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Embodiments of the present invention relate generally to videoprocessing and more specifically to video processing with multiplegraphics processing units.

2. Description of the Related Art

Oftentimes, video data must be processed prior to being displayed. Thereis a variety of video processing procedures that may be applied to videodata. Consider, for example, displaying data from a digital video disc(DVD) on a progressive display. If the content on the DVD has beenencoded in an interlaced format, then the video data needs to bede-interlaced before it can be displayed on the progressive display.Also, DVDs generally contain sub-title information. If a user wishes todisplay sub-title information, the sub-title text needs to be extractedfrom the video data and then composited into the video picture.De-interlacing and sub-title generation are only two examples of videoprocessing procedures. There are many other video processing proceduresthat may be applied to video data, such as edge-enhancement, picturescaling, color space conversion and the like. Further, video processingprocedures are generally not mutually exclusive. For example, if theuser wants to display video data from a DVD on a progressive display anddisplay sub-titles concurrently, then both the de-interlace and thesub-title generation procedures may be applied to the video data.However, since each procedure is executed on the same basic set of videodata, the different video processing procedure typically are applied tothe video data in series, rather than in a parallel fashion.

Video images are comprised of a sequence of video frames, where eachframe is comprised of two video fields. A typical frame rate used todisplay video frames is thirty frames per second (30 Hz). Therefore, thevideo processing procedures for the video data of a frame must executein less time than the time required to display one frame of video data(approximately 33 milliseconds). If the time required to process a frameof video data is greater than the time required to display a frame ofvideo data, then the processed video data cannot be displayed. Instead,previous video data is often shown in place of the current video data.This phenomenon is commonly referred to as “dropping” video frames andis quite undesirable because it results in a lack of motion smoothness,which is noticeable by the human eye, leading to poor video quality.

A graphics processing unit (GPU) may be configured to provide videoprocessing functionality within a video processing system. For example,the GPU may be configured to use a three-dimension (3D) pixel shader toprovide edge enhancement of the video data. Similarly, the GPU can beconfigured to implement other video processing procedures. Eachprocessing task requires a finite amount of time to complete. Since, asdescribed above, the frame rate limits the amount of time available toprocess each frame and the video processing procedures are generallyimplemented in series, the number and complexity of the video processingprocedures that may be executed on single GPU is limited. Exacerbatingthis problem is the fact that high definition video images requireprocessing up to six times more pixels than standard definition images.Increasing the pixel count increases the amount of time required toperform each processing procedure, thereby further limiting the numberof video processing procedures a single GPU can apply to a frame ofvideo data without exceeding the video frame time budget and, thus,increasing the chance of dropping the video frame.

As the foregoing illustrates, what is needed in the art is a way toincrease the video processing throughput of a video processing system sothat more processing procedures may be implemented on video data,including high definition video data, without increasing the incidenceof dropped frames.

SUMMARY OF THE INVENTION

One embodiment of the invention sets forth a system for processing videodata. The system includes a host processor, a first media processingdevice coupled to a first frame buffer and a second media processingdevice coupled to a second frame buffer. The first frame buffer isconfigured to store video data, and the first media processing device isconfigured to process a first portion of that video data. The secondframe buffer is configured to store a copy of the video data, and thesecond media processing device is configured to process a second portionof the video data. The two media processing devices are coupled togethervia a scalable bus. The scalable bus carries the second portion of thevideo data processed by the second media processing device to the firstmedia processing device where the data is combined with the firstportion of the video data processed by the first media processing deviceto produce a processed video frame.

One advantage of the disclosed system is that it provides a multi-mediaprocessing environment capable of processing video data more efficientlythan prior art, single-GPU video processing systems. As a result, videoframes may be processed in substantially less time relative to prior artsystems, and the number and complexity of the video processing commandsexecuted using the disclosed system may be substantially increasedrelative to prior art systems.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the presentinvention can be understood in detail, a more particular description ofthe invention, briefly summarized above, may be had by reference toembodiments, some of which are illustrated in the appended drawings. Itis to be noted, however, that the appended drawings illustrate onlytypical embodiments of this invention and are therefore not to beconsidered limiting of its scope, for the invention may admit to otherequally effective embodiments.

FIG. 1 is a conceptual illustration of the functional steps a videoprocessing system implements when processing a frame of video data usingmultiple media processing devices, according to one embodiment of theinvention;

FIG. 2 is a conceptual diagram of a computing device configured toimplement one or more aspects of the present invention;

FIG. 3 is a more detailed diagram of the computing device of FIG. 2,according to one embodiment of the invention;

FIG. 4 is a flow diagram of method steps implemented by the computingdevice of FIGS. 2 and 3 when processing a frame of video data, accordingto one embodiment of the invention;

FIG. 5A is a conceptual diagram illustrating a sequence of commands usedto implement step 402 of FIG. 4, according to one embodiment of theinvention;

FIG. 5B is a conceptual diagram illustrating a sequence of commands usedto implement step 403 of FIG. 4, according to one embodiment of theinvention; and

FIG. 5C is a conceptual diagram illustrating a sequence of commands usedto implement step 410 of FIG. 4, according to one embodiment of theinvention.

DETAILED DESCRIPTION

Among other things, the invention described herein enables two or moremedia processing devices to be implemented in a video processing systemto process frames of video data. By using multiple media processingdevices to process video frames, video processing throughput may beincreased, which allows more video processing procedures, and/or morecomplex procedures, to be applied to the video data without increasingthe incidence of dropped frames. Conversely, if the same number of videoprocessing procedures is applied to the video data using multiple mediaprocessing devices, the incidence of dropped frames decreases.

FIG. 1 is a conceptual illustration of the functional steps a videoprocessing system implements when processing a frame of video data usingmultiple media processing devices, according to one embodiment of theinvention. Persons skilled in the art will recognize that any systemconfigured to perform these functional steps in any order is within thescope of the invention. In particular, the embodiments of the videoprocessing system disclosed herein depict graphics processing units asthe media processing devices within the system. However, in alternativeembodiments, any type of video or media accelerator may be implemented.For example, the video processing system described herein could beimplemented with a device having one or more video processing enginesthat does not have any particular graphics processing capabilities.

The first functional step is step 102, where a master GPU receivesdecoded video data. In a multi-GPU system having two GPUs, a first GPUis designated the master GPU and a second GPU is designated the slaveGPU. The video data may be received from sources such as a tuner, adecoder, a storage device or the like. Video data is typically receivedone frame at a time and is stored in a frame buffer associated with themaster GPU. As is well-known, the frame buffer is a section of the GPUmemory. There are many ways that the video data may be received into theframe buffer of the master GPU such as by a direct memory access (DMA)or by the CPU writing the video data directly into the frame buffer.

In step 106, the frame buffer of the slave GPU is synchronized with theframe buffer of the master GPU, enabling the master GPU and the slaveGPU to simultaneously process the video data in their respective framebuffers. In step 107, the frame buffers of the master GPU and the slaveGPU are divided into a first and second portion. Each of the GPUs isthen configured to process video data residing in only a specifiedportion its respective frame buffer to increase operationalefficiencies. For example, if the first portion is the upper part ofeach frame buffer and the second portion is the lower part of each framebuffer, the master GPU may be configured to process only the video dataresiding within the first portion of the master frame buffer, while theslave GPU may be configured to process only the video data residingwithin the second portion of the slave frame buffer. In step 108, theGPUs provide inverse telecine processing. Oftentimes, video dataoriginates from film, which has a frame rate of twenty-four frames persecond. If the video data is to be displayed at a frame rate of thirtyframes per second, then a process commonly called “three-two pulldown”is applied to the film data so that film images from the slower framerate may be displayed as video images at the higher frame rate. Thisprocess may add undesired visual artifacts, especially if the displaydevice does not require the three-two pulldown process, as is the casefor progressive displays. In such cases, the inverse telecine processingstep returns the video data to the original film frame rate. Asdescribed in further detail herein, when the GPUs process the video data(in this step and in others), each GPU processes only a portion of thedata in its frame buffer, thereby substantially increasing theprocessing efficiency of the video processing system.

In step 110, the GPUs provide de-interlacing processing. De-interlacingis a well-known process that transforms interlaced video data into videodata more suitable for display on progressive display devices. There aremany well-known methods that may be used to implement de-interlacingfunctionality, such as vertical interpolation (commonly called “BOB”)and displaying both fields simultaneously (commonly called “WEAVE”).Persons skilled in the art will recognize that other, more advanced,de-interlacing schemes also may be implemented on the GPUs. In step 112,the GPUs provide sub-picture processing. Sub-picture processing istypically used to insert subtitles and DVD menus into the video data.Subtitle and DVD information is extracted from the video data and thencomposited into the visible portion of the video data. In step 114, theGPUs provide edge enhancement processing. Sometimes the video data hasbeen processed in a manner that makes the edges of objects in the videoframe appear soft. Edge enhancement processing is commonly used toaugment the visual appearance of the edges. In step 116, the GPUsprovide picture scaling processing. In this step, the video data may bescaled to a greater or lesser resolution. This procedure is typicallyperformed when the resolution of the video display differs from theresolution of the source of the video data. In step 118, the GPUsprovide color space conversion processing. The color space requirementsof the display device may differ from the color space representation ofthe video data. For example, video data is often represented in the YUVcolor space; however, typical display devices operate in the RGB colorspace. In step 120, the GPUs provide LCD overdrive processing tocompensate for undesirable LCD display characteristics.

Once the video data has been processed, the processed data residing ineach GPU frame buffer is combined for display. In step 122, the GPUs aresynchronized since the GPUs may complete the video processing atdifferent times. As described in further detail herein, the master andslave GPUs are synchronized through a process of reading and writingsemaphores. In step 124, a software driver swaps the frame buffers.Video processing is often carried out with a well-known technique calleddouble buffering. Double buffering uses two frame buffers to store andprocess video data. A GPU uses a first frame buffer to process a currentframe of video data, while previously processed video data is read froma second frame buffer and transmitted to a display. After the last pixelof the second frame buffer is transmitted to the display, the two framebuffers are “swapped” so that the processed data from the first framebuffer may be read and transmitted to the display device, while the GPUprocesses a new frame of video data in the second frame buffer. Finally,in step 126, the video data is displayed on a display.

Aspects of these functional steps are described in further detail belowin conjunction with FIG. 4. Alternative embodiments may include othervideo processing steps, such as noise filtering, color enhancement orthe like. Still other embodiments may omit some video processing steps.For example, if the video data is not displayed on an LCD display, thenthe LCD overdrive processing step 120 may be skipped. Persons skilled inthe art also will understand that the concept of double buffering,described above in step 124, may be extended to triple buffering, wherethree buffers instead of two buffers are used. Triple buffering isespecially useful when video decoding is performed. In such cases, onebuffer is typically used for decode, one buffer is used forpost-processing, and the third buffer is used for display, where allthree functions run concurrently.

FIG. 2 is a conceptual diagram of a computing device 200 configured toimplement one or more aspects of the current invention. The computingdevice 200 includes, without limitation, a central processing unit (CPU)202, system memory 204, a first graphics subsystem 208 and a secondgraphics subsystem 210. The CPU 202 is coupled to the system memory 204,which is used to store data and programs, such as a driver 206 that canconfigure the graphics subsystems 208 and 210 to provide the desiredvideo processing functionality.

The first graphics subsystem 208 includes a first GPU 212 coupled to afirst GPU memory 214, which is configured to store GPU instructions anddata, such as video data. As described above, video data is stored in adoubled-buffered frame buffer that includes a first frame buffer 215 anda second frame buffer 216. For purposes of discussion only, it isassumed that a portion of the current video frame is being processed inthe first frame buffer 215 and a portion of the video frame currentlybeing transmitted to the display device is being read from the secondframe buffer 216. The first GPU 212 also includes a digital to analogconverter (DAC) 218, which is used to transmit processed video to adisplay. As shown, the first GPU 212 is designated as the master GPU.Typically, the master GPU 212 displays the processed video data. Inalternative embodiments, video data may be transmitted via other displayinterfaces included in the first GPU 212, such as transition minimizeddifferential signaling (TMDS) interface, a serial digital interface(SDI) or the like.

The second graphics subsystem 210 includes a second GPU 222 and a secondGPU memory 224. In one embodiment, the second graphics subsystem 210 issubstantially similar to the first graphics subsystem 208. The secondGPU 222 is coupled to the second GPU memory 224, which is configured tostore GPU instructions and data, such as video data. Again, video datais stored in double-buffered frame buffer that includes a third framebuffer 225 and a fourth frame buffer 226. For purposes of discussiononly, it is assumed that a portion of the current video frame is beingprocessed in the third frame buffer 225 and a portion of the video framecurrently being transmitted to the display device is being read from thefourth frame buffer 226. As shown, the second GPU 222 is designated asthe slave GPU. Since the slave GPU 222 is typically not used to transmitthe processed video data to a display, a DAC 220 within the slave GPU222 may be left unconnected.

The first GPU 212 is coupled to the second GPU 222 by a GPU bus 250. TheGPU bus 250 is a scalable bus used by the second GPU 222 to transmitprocessed video data to the first GPU 212. For example, in oneembodiment, the GPU bus 250 may be implemented using the NVIDIA SLI™multi-GPU technology. Further, as previously described herein, thecomputing device 200 may be implemented using any type of video or mediaaccelerator. In alternative embodiments, therefore, the GPU bus 250 maybe any type technically feasible scalable bus that transmits processedvideo data between media processing devices.

In operation, the driver 206 generates a stream of commands called a“push buffer.” When executed, the commands in the push buffer enablemulti-GPU processing of video data stored in the frame buffers of thefirst GPU 212 and the second GPU 222. First, video data is copied fromthe first frame buffer 215 to the third frame buffer 225. Second, asdescribed above, the frame buffers are split into first and secondportions (which effectively divides the video data into first and secondportions), and the first GPU 212 is configured to process the firstportion of the video data (i.e., the video data residing in the firstportion of the first frame buffer 215), and the second GPU 222 isconfigured to process only the second portion of the video data (i.e.,the video data residing in the second portion of the third frame buffer225). Third, the video data residing in the first portion of the firstframe buffer 215 and the second portion of the third frame buffer 225 isprocessed. Fourth, after the video data is processed, the first GPU 212and the second GPU 222 are synchronized, and then the first frame buffer215 and the third frame buffer 225 are swapped with the second framebuffer 216 and the fourth frame buffer 226, respectively. Fifth, theprocessed video data is transmitted from the second portion of the thirdframe buffer 225 to the first GPU 212. This data is combined with theprocessed video data from the first portion of the first frame buffer215 to produce a processed video frame that is then transmitted fordisplay. These operations are described in greater detail below inconjunction with FIG. 4.

In alternative embodiments, more than two graphics subsystems may beused in the computing device 200. In such a configuration, there is asingle master GPU and two or more slave GPUs. The slave GPUs are coupledto the master GPU via the GPU bus 250. The video data processed by theslave GPUs is transmitted to the master GPU through the GPU bus 250. Themaster GPU transmits the combined processed video data to the displaydevice.

FIG. 3 is a more detailed diagram of the computing device 200 of FIG. 2,according to one embodiment of the invention. In particular, the masterGPU 212 and the slave GPU 222 and the GPU memories 214 and 224 aredescribed in more detail herein. As shown, the master GPU 212 includes,without limitation, a video processor 310, a three-dimensional (3-D)processor 314, a two-dimensional (2-D) processor 318, a host interface302, a memory interface 330 and a GPU bus interface 340. The hostinterface 302 is coupled to the video processor 310, the 3-D processor314 and the 2-D processor 318. The host interface 302 receives the pushbuffer commands from the CPU 202 (and the driver 206) and configures thevideo processor 310, the 3-D processor 314 and the 2-D processor 318 toprocess video data according to the push buffer commands. The videoprocessor 310, the 3-D processor 314 and the 2-D processor 318 arefurther coupled to the memory interface 330. The memory interface 330enables the video processor 310, the 3-D processor 314 and the 2-Dprocessor 318 to access video data stored in the GPU memory 214.

Similarly, the slave GPU 222 includes, without limitation, a videoprocessor 312, a 3-D processor 316, a 2-D processor 320, a hostinterface 304, a memory interface 332 and a GPU bus interface 342.Again, the host interface 304 is coupled to the video processor 312, the3-D processor 316 and the 2-D processor 320. The host interface 304receives the push buffer commands from the CPU 202 (and the driver 206)and configures the video processor 312, the 3-D processor 316 and the2-D processor 320 to process video data according to the push buffercommands. The video processor 312, the 3-D processor 316 and the 2-Dprocessor 320 are further coupled to the memory interface 332. Thememory interface 332 enables the video processor 312, the 3-D processor316 and the 2-D processor 320 to access video data stored in the GPUmemory 224.

There are many well-known ways in which video processing functionalitymay be provided by processing units within the master GPU 212 and theslave GPU 222. For example, the video processors 310 and 312 may beconfigured to provide video processing functionality such as picturescaling. The 3-D processors 314 and 316 may be configured to uses theincluded pixel shaders to provide video processing functionality such asedge enhancement. The 2-D processors 318 and 320 may be configured toprovide video processing functionality such as a memory blit to copy thevideo data from the master GPU memory 214 to the slave GPU memory 224. Amemory blit is a common 2-D process that copies the contents of a blockof memory from one location to another.

As also shown, the GPU memory 214 includes a first frame buffer 215 anda second frame buffer 216, and the GPU memory 224 includes a third framebuffer 225 and a fourth frame buffer 226. Again, for purposes ofdiscussion only, it is assumed that the master GPU 212 and slave GPU 222are processing the video data residing in the first and third framebuffers 215, 225, while the previously processed video data resides inthe second and fourth frame buffers 216, 226. Since, as describedearlier in FIG. 1, decoded video is written into the first frame buffer215 of the master GPU 212, the driver 206 configures the master GPU 212to copy the contents of the first frame buffer 215 to the third framebuffer 225. The driver 206 also provides commands to synchronize themaster GPU 212 and the slave GPU 222 so that the video data in the firstframe buffer 215 and the third frame buffer 225 may be simultaneouslyprocessed. After the video data is copied and the GPUs are synchronized,the driver 206 divides the video data in the frame buffers 215 and 225each into a first portion and a second portion. Each of the master GPU212 and the slave GPU 222 processes a portion of the video data in itsrespective frame buffer. For example, driver 206 may configure themaster GPU 212 to process the video data residing in a first portion 350of the first frame buffer 215. Similarly, the driver 206 may configurethe slave GPU 220 to process the video data residing in a second portion354 of the third frame buffer 225. After the current video frame isprocessed, the first frame buffer 215 is swapped with the second framebuffer 216, and the third frame buffer 225 is swapped with the fourthframe buffer 226. Thus, when processing the next video frame, the masterGPU 212 processes the video data residing in a first portion 352 of thesecond frame buffer 216, and the slave GPU 222 processes the video dataresiding in a second portion 356 of the fourth frame buffer 226.Processing efficiency is increased since each GPU is processes only aportion of the video data comprising the video frame.

After the master GPU 212 and the slave GPU 222 complete the videoprocessing procedures on the video data, but before the frame buffersare swapped, the driver 206 synchronizes the master GPU 212 and theslave GPU 222. By synchronizing the GPUs 212 and 222, the driver 206ensures that both the master GPU 212 and the slave GPU 222 havecompleted processing their respective video data before swapping theframe buffers. This synchronization step is important because, amongother things, if one of the GPUs has not yet completed processing itsportion of the video data when the system attempts to swap the framebuffers and combine the video data, then the video frame being processedmay be dropped.

The GPU bus interfaces 340 and 342 enable two or more GPUs to be coupledtogether through the GPU bus 250. Again, as described herein, inalternative embodiments, the GPU bus 250 may be any type of technicallyfeasible scalable bus used to transmit processed video data betweenwhatever types of media processing devices are implemented in thecomputing device 200. Thus, alternative embodiments of GPU businterfaces 340 and 342, may be any technically feasible interfaces tothe scalable bus used to transmit processed video data between mediaprocessing devices. The GPU bus interface 342 of the slave GPU 222 iscoupled to the memory interface 332 and the DAC 220. The GPU businterface 340 of the master GPU 212 is also coupled to the memoryinterface 332 and the DAC 218. The driver 206 configures the GPU businterface 342 of the slave GPU 220 to transmit the processed video datafrom the relevant portions (e.g., the second portion 354 and the secondportion 356) of the third and fourth frame buffers 225 and 226 to themaster GPU 212. Likewise, the driver 206 configures the GPU businterface 340 of the master GPU 212 to receive the processed video datafrom the slave GPU 220 and to transmit that processed video data to theDAC 218.

The DAC 218 of the master GPU 212 is coupled to memory interface 330 aswell as to a display device (not shown) or a memory element (not shown)where processed video frames are stored until they are displayed. TheDAC 220 of the slave GPU is similarly coupled to memory interface 332,but, typically, the DAC 220 of the slave GPU 222 is not connected to anydisplay or related memory element. The DAC 218 of the master GPU 212 isconfigured to combine the processed video data within the relevantportions (e.g., the first portion 350 and the first portion 352) of thefirst and second frame buffers 215 and 216, respectively, with theprocessed video data received by the GPU bus interface 340 from theslave GPU 222 to create a processed video frame. The DAC 218 is furtherconfigured to transmit the processed video frame to the display deviceor related memory element, as the case may be.

FIG. 4 is a flow diagram of method steps implemented by the computingdevice of FIGS. 2 and 3 when processing a frame of video data, accordingto one embodiment of the invention. Persons skilled in the art willrecognize that any system configured to perform the method steps in anyorder is within the scope of the invention. Further, the methoddescribed herein may be repeated for each frame of video data to beprocessed.

The method begins in step 402, where the driver 206 configures themaster GPU 212 to copy the video data from the first frame buffer 215 ofthe master GPU 212 to the third frame buffer 225 of the slave GPU 222.For the purposes of discussion only, it is assumed that the master GPU212 and the slave GPU 222 are processing the current video frame in thefirst and third frame buffers 215 and 225, respectively, and that thepreviously processed video frame is being transmitted from the secondand fourth frame buffers 216 and 226 for display. The master GPU 212 andthe slave GPU 222 are then synchronized by series of semaphore commandsin the push buffer. The semaphore commands prevent the slave GPU 222from executing subsequent push buffer commands while the master GPU 212copies the video data from the first frame buffer 215 to the third framebuffer 225. Effectively, the slave GPU 222 is forced to “wait” for themaster GPU 212 to finish copying the video data. Once the video data hasbeen copied, both the master GPU 212 and the slave GPU 222 are able toresume executing commands in the push buffer and, thus, aresynchronized. This copy and synchronization step is described in greaterdetail below in conjunction with FIG. 5A.

In step 403, the driver 206 configures the master GPU 212 and the slaveGPU 222 to divide the video data residing in the first frame buffer 215and the third frame buffer 225 into first and second portions. Themaster GPU 212 and the slave GPU 222 are then configured to processdifferent portions of the video data. For example, the master GPU 212may be configured to process the video data residing in the firstportion 350 of the first frame buffer 215, and the slave GPU 222 may beconfigured to process the video data residing in the second portion 354of the third frame buffer 225. As a result, the master GPU 212 and slaveGPU 222 share the processing of the current video frame, therebyincreasing the processing efficiency of the computing device 200. Thestep of dividing the video data into different portions is described ingreater detail below in conjunction with FIG. 5B.

In step 404, the driver 206 configures the video processors 310 and 312.Such a configuration may, for example, configure the video processors310 and 312 to scale the video data. In step 406, the driver 206configures the 3-D processors 314 and 316. Such a configuration may, forexample, configure the 3-D processors 314 and 316 to providede-interlacing functionality, edge enhancement or other video processingfunctionality. In step 408, the driver 206 configures the 2-D processors318 and 320. Such a configuration may, for example, configure the 2-Dprocessors 318 and 320 to insert sub-title information or provide menusscreens typical in DVD applications. In other embodiments, the masterGPU 212 and the slave GPU 222 may be configured to provide a greater orlesser amount of video processing than is described in steps 404, 406and 408 or different types of video processing.

In step 410, the driver 206 synchronizes the master GPU 212 and theslave GPU 222. Since each GPU may process video data at different rates,this step ensures that both GPUs have finished processing the video datain the frame buffers 215 and 225 before the frame buffers are swappedand the processed video data is combined to produce the processed videoframe. Again, as described in greater detail below in conjunction withFIG. 5C, a series of semaphore commands in the push buffer are used tosynchronize the master GPU 212 and the slave GPU 222.

In step 412, the driver 206 swaps the frame buffers. Continuing theexample set forth in step 402, after the final pixel from each of thefirst portion 352 of the second frame buffer 216 and the second portion356 of the fourth frame buffer 226 is displayed, the driver 206 swapsthe first frame buffer 215 and the second frame buffer 216 as well asthe third frame buffer 225 and fourth frame buffer 226 so that theprocessed video data in the first portion 350 of the first frame buffer215 and the second portion 354 of the third frame buffer 225 may betransmitted to the DAC 218 for display. The next frame of video data tobe processed is then stored in the second frame buffer 216 and thefourth frame buffer 226.

In step 414, the processed video data is combined into a processed videoframe and transmitted to a display device (or related memory element) bythe DAC 218. Again, continuing the example, the processed video datafrom the first portion 350 of the first frame buffer 215 and from thesecond portion 354 of the third frame buffer 225 is combined by the DAC218 to produce the processed video frame.

FIG. 5A is a conceptual diagram illustrating a sequence of commands 502used to implement step 402 of FIG. 4, according to one embodiment of theinvention. As shown, the commands 502 form a portion of a push bufferthat is assembled by the driver 206 to configure the master GPU 212 andthe slave GPU 222 to process video data. Specifically, the commands 502cause the master GPU 212 to copy video data residing in the first framebuffer 215 to the third frame buffer 225 and also cause thesynchronization of the master GPU 212 and the slave GPU 222 to enablethe master GPU 212 and the slave GPU 222 to simultaneously processportions of the video data residing in the first frame buffer 215 andthe third frame buffer 225.

A set sub-device mask (SSDM) command 504 sets a sub-device mask to 01,enabling only the master GPU 212 to execute subsequent commands in thepush buffer. The argument to the SSDM command determines which GPU isconfigured to execute subsequent commands in the push buffer. In oneembodiment, the argument is a two-bit bit field, where each bit withinthe bit field corresponds to one of the two GPUs. If, for example, thefirst bit corresponds to the master GPU 212 and the second bitcorresponds to the slave GPU 222, then an SSDM 01 command wouldconfigure the master GPU 212 to execute subsequent commands in the pushbuffer commands, while the slave GPU 222 ignores the subsequentcommands. An SSDM 11 command would configure both the master GPU 212 andthe slave GPU 222 to execute subsequent commands in the push buffer. Thenext command is a copy command 506, which directs the master GPU 212 tocopy the contents of the first frame buffer 215 to the third framebuffer 225. The execution of this command provides the master GPU 212and the slave GPU 222 access to the same video data.

A release semaphore command 508 causes the master GPU 212 to release asemaphore. The driver uses semaphores to synchronize the master GPU 212and the slave GPU 222 to enable the GPUs to simultaneously processportions of the video data in the first frame buffer 215 and the thirdframe buffer 225. A semaphore is a pointer to a specific address insystem memory. A semaphore may be released or acquired. When a GPUexecutes a release semaphore command, the GPU writes a specific value tothe memory location associated with the semaphore. When a GPU executesan acquire semaphore command, the GPU reads the memory locationassociated with the semaphore and compares the value of that memorylocation with the value reflected in the acquire semaphore command. Thetwo values not matching indicates that the semaphore associated with theacquire semaphore command has not yet been released. If there is nomatch, the GPU executing the acquire semaphore command continues readingthe memory location associated with the semaphore until a match isfound. Consequently, the GPU executing the acquire semaphore commanddoes not execute any additional push buffer commands until a match isfound. For example, assume that a first GPU is directed to release asemaphore having a value of 42 and then a second GPU is directed toacquire the semaphore having a value of 42. The second GPU will continuereading the system memory location associated with the semaphore untilthat memory location has a value of 42. Importantly, the second GPU willnot execute the next push buffer command until the memory location hasvalue of 42, and the memory will have a value of 42 only when the firstGPU releases the semaphore having a value of 42.

An SSDM command 510 sets the sub-device mask to 10, enabling only theslave GPU 222 to execute subsequent commands in the push buffer. Anacquire semaphore command 512 directs the slave GPU 222 to acquire thesemaphore released by the master GPU 212 in response to the releasesemaphore command 508. As described above, since the slave GPU 222 willnot execute any additional push buffer commands until the master GPU 212releases the semaphore, and the master GPU 212 does not release thesemaphore until the copy command 506 has been fully executed, the slaveGPU 222 is forced to “wait” for the master GPU 212 to copy the videodata from the first frame buffer 215 to the third frame buffer 225before executing any additional push buffer commands. Thus, the masterGPU 212 and the slave GPU 222 are “synchronized” once the semaphore isreleased by the master GPU 212 and acquired by the slave GPU 222. AnSSDM command 514 sets the sub-device mask to 11, enabling both GPUs toexecute the next command in the push buffer.

FIG. 5B is a conceptual diagram illustrating a sequence of commands 530used to implement step 403 of FIG. 4, according to one embodiment of theinvention. As shown, the commands 530 form a portion of the push bufferthat is assembled by the driver 206 to configure the master GPU 212 andthe slave GPU 222 to process video data. Specifically, the commands 530divide the video data into a first portion and a second portion. Themaster GPU 212 processes the first portion of the video data, and theslave GPU 222 processes the second portion of the video data. Thus, themaster GPU 212 and the slave GPU 222 are able to simultaneously processthe video data, leading to enhanced processing efficiencies relative toprior art systems.

An SSDM command 538 sets the sub-device mask to 01, enabling only themaster GPU 212 to execute subsequent commands in the push buffer. A setclip rectangle command 540 sets the clip rectangle for the master GPU212. The driver 206 uses set clip rectangle commands to define theportion of the video data that each GPU processes. In one embodiment,the set clip rectangle command defines the boundaries of the portion ofvideo data that a particular GPU processes by specifying four corners ofthe frame buffer coupled to that GPU. The GPU is then configured toprocess video data that resides within those four corners of the framebuffer. For example, if the driver 206 divided a frame bufferapproximately in half, the upper portion of the frame buffer could bespecified by the set clip rectangle command with corner pixelcoordinates (0,0), (w,0), (h/2,0) and (h/2,w), where h is the height ofthe frame buffer and w is the width of the frame buffer. The lowerportion of the frame buffer could be specified by the set clip rectanglecommand with corner pixel coordinates (h/2,0), (h/2,w), (h,0) and (h,w).Thus, the clip rectangle command 540 specifies four corners of the framebuffer 215. The video data within these four corners constitutes theportion of the video data processed by the master GPU 212.

Since a larger clip rectangle includes relatively more video data than asmall clip rectangle, the driver 206 may control the amount ofprocessing performed by each GPU by controlling the amount of video datain the specified clip rectangles. In alternative embodiments, theportion of the video data that a given GPU processes may be specified bypixel location within the frame, by line number or by any othertechnically feasible means.

An SSDM command 542 sets the sub-device mask to 10, enabling only theslave GPU 222 to execute subsequent commands in the push buffer. A setclip rectangle command 544 sets the clip rectangle for the slave GPU 222by specifying four corners of the frame buffer 225. The video datawithin these four corners constitutes the portion of the video dataprocessed by the slave GPU 222. An SSDM command 546 sets the sub-devicemask to 11, enabling both the master GPU 212 and the slave GPU 222 toexecute subsequent commands in the push buffer. As previously describedherein, if the next command in the push buffer is a video processingcommand, the master GPU 212 executes that command on the portion of thevideo data in the first frame buffer 215 within the clip rectanglespecified in the set clip rectangle command 540. Likewise, the slave GPU222 executes that command on the portion of the video data in the thirdframe buffer 225 within the clip rectangle specified in the set cliprectangle command 544.

FIG. 5C is a conceptual diagram illustrating a sequence of commands 560used to implement step 410 of FIG. 4, according to one embodiment of theinvention. As shown, the commands 560 form a portion of the push bufferthat is assembled by the driver 206 to configure the master GPU 212 andthe slave GPU 222 to process video data. Specifically, the commands 560synchronize the master GPU 212 and the slave GPU 222 after the videodata in the first frame buffer 215 and the third frame buffer 225 hasbeen processed. As described above in FIG. 5B, in a multi-GPU system,the master GPU 212 and the slave GPU 222 process different portions ofthe video data. As a consequence, the master GPU 212 and the slave GPU222 may finish processing their respective portions of the video data atslightly different times. Synchronizing the master GPU 212 and the slaveGPU 222 ensures that each GPU has completed processing the video data inthe first frame buffer 215 and the third frame buffer 225, respectively,before the video data processed by the slave GPU 222 is transmitted tothe master GPU 212 and combined with the video data processed by themaster GPU 212 to produce the processed video frame. Thissynchronization step is important because, among other things, if one ofthe GPUs has not yet completed processing its portion of the video datawhen the system attempts to combine the video data, then that videoframe may be dropped.

An SSDM command 564 sets the sub-device mask to 01, enabling only themaster GPU 212 to respond to subsequent commands in the push buffer. Anacquire semaphore command 566 directs the master GPU to acquire asemaphore. An SSDM command 568 sets the sub-device mask to 10, enablingonly the slave GPU 222 to execute subsequent commands in the pushbuffer. A release semaphore command 570 directs the slave GPU 222 torelease the semaphore specified in the acquire semaphore command 566. Aspreviously described herein, since the master GPU 212 does not executeany additional push buffer commands until the slave GPU 222 releases thesemaphore, the GPUs are synchronized once the slave GPU 222 releases thesemaphore and the master GPU 212 acquires the semaphore because, at thispoint, both GPUs are able to execute the next command in the pushbuffer. An SSDM command 572 sets the sub-device mask to 11, enablingboth GPUs to execute subsequent command in the push buffer.

Persons skilled in the art will recognize that the above commands may beused to configure the master GPU 212 and the slave GPU 222 such thatonly one of the GPUs executes certain processing commands on itsrespective portion of the video data. For example, if subtitles need tobe added to the bottom of a video frame, SSDM commands may be used toenable one GPU, while disabling the other. The enabled GPU would be theGPU responsible for processing the bottom portion of the video data.That GPU would then execute the processing commands required to add thedesired subtitles. Since the other GPU would be disabled, no processingwould be performed on the upper portion of the video data while thesubtitles are added.

One advantage of the disclosed systems and methods is that they providea multi-GPU video processing system that processes video data moreefficiently than prior art, single-GPU systems. As a result, videoframes may be processed in substantially less time relative to prior artsystems, and the number and complexity of the video processing commandsexecuted using the disclosed system may be substantially increasedrelative to prior art systems.

While the foregoing is directed to embodiments of the present invention,other and further embodiments of the invention may be devised withoutdeparting from the basic scope thereof, and the scope thereof isdetermined by the claims that follow.

1. A system for processing video data, the system comprising: a hostprocessor; a first media processing device coupled to a first framebuffer, wherein the first frame buffer is configured to store videodata, and the first media processing device is a master media processingdevice configured to process a first portion of the video data; a secondmedia processing device coupled to a second frame buffer, wherein thesecond frame buffer is configured to store a copy of the video data, andthe second media processing device is a slave media processing deviceconfigured to process a second portion of the video data, wherein thefirst media processing device is coupled to the second media processingdevice via a scalable bus, wherein a driver generates a push buffercomprising a plurality of commands for processing the video data thatinclude: one or more commands for copying the video data stored in thefirst frame buffer to the second frame buffer, a first set sub-devicemask (SSDM) command that configures the first media processing deviceand the second media processing device such that only the first mediaprocessing device executes commands in the push buffer subsequent to thefirst SSDM command, a copy command that causes the first mediaprocessing device to copy the video data stored in the first framebuffer to the second frame buffer, a release semaphore command thatcauses the first media processing device to release a semaphore, asecond SSDM command that configures the first media processing deviceand the second media processing device such that only the second mediaprocessing device executes subsequent commands in the push buffersubsequent to the second SSDM command, an acquire semaphore command thatcauses the second media processing device to acquire the semaphorereleased by the first media processing device, and a third SSDM commandthat configures the first media processing device and the second mediaprocessing device such that both the first media processing device andthe second media processing device execute commands in the push buffersubsequent to the third SSDM command, and wherein after the processingof both the first and second portion of the video data is completed, thesecond media processing device is configured to transmit the processedsecond portion of the video data to the first media processing devicevia the scalable bus.
 2. The system of claim 1, wherein the first mediaprocessing device and the second media processing device aresynchronized after the second media processing device acquires thesemaphore released by the first media processing device.
 3. The systemof claim 1, wherein the plurality of commands includes commands fordefining the first portion of the video data and the second portion ofthe video data.
 4. The system of claim 3, wherein the plurality ofcommands includes a first set clip rectangle command that defines thefirst portion of the video data and a second set clip rectangle commandthat defines the second portion of the video data.
 5. The system ofclaim 4, wherein the plurality of commands includes at least one videoprocessing command that the first media processing device executes onthe first portion of the video data and the second media processingdevice executes on the second portion of the video data.
 6. The systemof claim 1, wherein the plurality of commands includes commands forsynchronizing the first media processing device and the second mediaprocessing device, once the video data has been processed.
 7. The systemof claim 6, wherein the plurality of commands includes an acquiresemaphore command that causes the first media processing device toacquire a semaphore released by the second media processing device and arelease semaphore command that causes the second media processing deviceto release the semaphore.
 8. The system of claim 7, wherein the firstmedia processing device and the second media processing device aresynchronized after the first media processing device acquires thesemaphore released by the second media processing device.
 9. The systemof claim 1, wherein the second portion of the video data processed bythe second media processing device is transmitted to the first mediaprocessing device via the scalable bus and combined with the firstportion of the video data processed by the first media processing deviceto produce a processed video frame.
 10. The system of claim 9, whereineach of the first frame buffer and the second frame buffer is swappedwith another frame buffer before the second portion of the video data istransmitted to the first media processing device via the scalable bus.11. The system of claim 1, wherein each of the first media processingdevice and the second media processing device is a graphics processingunit.
 12. A method for processing video data using multiple mediaprocessing devices, the method comprising: copying video data stored ina first frame buffer that is coupled to a master media processing deviceto a second frame buffer that is coupled to a slave media processingdevice; defining a first portion and a second portion of the video data;executing at least one video processing command on the first portion ofthe video data stored in the first frame buffer; executing the at leastone video processing command on the second portion of the video datastored in the second frame buffer; synchronizing the master mediaprocessing device and the slave media processing device prior totransmitting the processed second portion of the video data, by:executing a first SSDM that configures the master media processingdevice and the slave media processing device such that only the mastermedia processing device executes commands in a push buffer subsequent tothe first SSDM command, executing an acquire semaphore command thatcauses the master media processing device to acquire a semaphorereleased by the slave media processing device, executing a second SSDMcommand that configures the master media processing device and the slavemedia processing device such that only the slave media processing deviceexecutes commands in the push buffer subsequent to the second SSDMcommand, executing a release semaphore command that causes the slavemedia processing device to release the semaphore, and executing a thirdSSDM command that configures the master media processing device and theslave media processing device such that both the master media processingdevice and the slave media processing device execute commands in thepush buffer subsequent to the third SSDM command; and after theprocessing of both the first and second portion of the video data iscompleted, transmitting the second portion of the video data processedin the second frame buffer to the master media processing device via ascalable bus.
 13. The method of claim 12, further comprising the step ofswapping each of the first frame buffer and the second frame buffer withanother frame buffer prior to transmitting the processed second portionof the video data.
 14. The method of claim 12, wherein the step ofcopying comprises executing a first set sub-device mask SSDM commandthat configures the master media processing device and the slave mediaprocessing device such that only the master media processing deviceexecutes commands in a push buffer subsequent to the first SSDM command,executing a copy command that causes the master media processing deviceto copy the video data stored in the first frame buffer to the secondframe buffer, executing a release semaphore command that causes themaster media processing device to release a semaphore, executing asecond SSDM command that configures the master media processing deviceand the slave media processing device such that only the slave mediaprocessing device executes commands in the push buffer subsequent to thesecond SSDM command, executing an acquire semaphore command that causesthe slave media processing device to acquire the semaphore released bythe master media processing device, and executing a third SSDM commandthat configures the master media processing device and the slave mediaprocessing device such that both the master media processing device andthe slave media processing device execute commands in the push buffersubsequent to the third SSDM command.
 15. The method of claim 12,wherein the step of defining a first portion and a second portioncomprises executing a first SSDM command that configures the mastermedia processing device and the slave media processing device such thatonly the master media processing device executes commands in a pushbuffer subsequent to the first SSDM command, executing a first set cliprectangle command that defines the first portion of the video data,executing a second SSDM command that configures the master mediaprocessing device and the slave media processing device such that onlythe slave media processing device executes commands in the push buffersubsequent to the second SSDM commands, executing a second set cliprectangle command that defines the second portion of the video data, andexecuting a third SSDM command that configures the master mediaprocessing device and the slave media processing device such that boththe master media processing device and the slave media processing deviceexecute commands in the push buffer subsequent to the third SSDMcommand.
 16. The method of claim 12, wherein each of the first mediaprocessing device and the second media processing device is a graphicsprocessing unit.